Information retrieval systems are widely used by users to search for information on a given subject. Web-based search systems are an example of one type of information retrieval system. Users submit a query to the web-based search system and obtain a list of results comprising links to documents that are relevant to the entered query. The list of results returned may be ranked by the search system such that results which are considered more relevant to the entered query are presented towards the top of the list and results which are less relevant are presented lower down on the list of results. Machine learning algorithms may be used to perform the ranking of results, where the algorithms are trained offline using labeled data. These labels are typically generated by human reviewers who make an assessment of the relevance of a particular label to a particular query.
The embodiments described below are not limited to implementations which solve any or all of the disadvantages of known information retrieval systems.